Skip to content

modify emit() of TopK to emit on batch_size rather than batch_size-1 - #10030

Merged
Dandandan merged 1 commit into
apache:mainfrom
JasonLi-cn:feature/optimize_topk
Apr 10, 2024
Merged

Dandandan merged 1 commit into
apache:mainfrom
JasonLi-cn:feature/optimize_topk

Conversation

@JasonLi-cn

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Closes #.

Rationale for this change

Simply modify the logic of "break into record batches as needed". This modification avoid generating an empty RecordBatch where the batch.num_rows() eq batch_size.

https://github.com/apache/arrow-datafusion/blob/03d8ba1f0d94bac6bb8bb33e95f00f9f6fb5275a/datafusion/physical-plan/src/topk/mod.rs#L210-L219

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

@alamb alamb left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

makes sense to me -- thank you @JasonLi-cn

@alamb alamb changed the title modify emit() of TopK modify emit() of TopK to emit on batch_size rather than batch_size-1 Apr 10, 2024
@alamb

alamb commented Apr 10, 2024

Copy link
Copy Markdown
Contributor

(BTW @JasonLi-cn I am not sure what is inspiring all your work recently to optimize DataFusion but it is much appreciated 🙏 )

@Dandandan
Dandandan merged commit 69595a4 into apache:main Apr 10, 2024
@Dandandan

Copy link
Copy Markdown
Contributor

Thanks @JasonLi-cn

@JasonLi-cn

JasonLi-cn commented Apr 11, 2024 •

Copy link
Copy Markdown
Contributor Author

(BTW @JasonLi-cn I am not sure what is inspiring all your work recently to optimize DataFusion but it is much appreciated 🙏 )

I have been using DataFusion and arrow-rs since 2021, but most of my work has been on developing TableScan or upper-level applications, as well as some optimizations for specific scenarios. My current work is mainly focused on performance optimization (which is also my interest) and feature enhancement, and I think it is necessary to contribute the improvements directly to the community, which is beneficial for the healthy development of both DataFusion and our own projects.
My PR may not be perfect, and I also thank all the community friends for their suggestions, from which I have learned a lot 🙏.

@alamb

alamb commented Apr 11, 2024

Copy link
Copy Markdown
Contributor

My current work is mainly focused on performance optimization (which is also my interest) and feature enhancement, and I think it is necessary to contribute the improvements directly to the community

100%

My PR may not be perfect, and I also thank all the community friends for their suggestions, from which I have learned a lot 🙏.

I think this describes everyone here (including myself!) I also learn a lot from interacting with other developers and working with the communnity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants